{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d0831f1b",
   "metadata": {},
   "source": [
    "# 机器学习调参练习"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62875267",
   "metadata": {},
   "source": [
    "在机器学习中，超参数是指无法从数据中学习而需要在训练前提供的参数。机器学习模型的性能在很大程度上依赖于寻找最佳超参数集。\n",
    "\n",
    "超参数调整一般是指调整模型的超参数，这基本上是一个非常耗时的过程。在本文中，我们将和你一起研习 3 种最流行的超参数调整技术：\n",
    "\n",
    "- **网格搜索**\n",
    "\n",
    "- **随机搜索**\n",
    "\n",
    "- **贝叶斯搜索**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "495f25f8",
   "metadata": {},
   "source": [
    "其实还有第零种调参方法，就是手动调参，因为简单机械，就不在本文讨论范围内。\n",
    "\n",
    "为方便阅读，列出本文的结构如下：\n",
    "\n",
    "1.获取和准备数据 \n",
    "\n",
    "2.网格搜索 \n",
    "\n",
    "3.随机搜索 \n",
    "\n",
    "4.贝叶斯搜索 \n",
    "\n",
    "5.写在最后\n",
    "\n",
    "## 获取和准备数据\n",
    "\n",
    "\n",
    "\n",
    "为演示方便，本文使用内置乳腺癌数据来训练**支持向量分类**（SVC）。可以通过`load_breast_cancer`函数获取数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d9ccf734",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean radius</th>\n",
       "      <th>mean texture</th>\n",
       "      <th>mean perimeter</th>\n",
       "      <th>mean area</th>\n",
       "      <th>mean smoothness</th>\n",
       "      <th>mean compactness</th>\n",
       "      <th>mean concavity</th>\n",
       "      <th>mean concave points</th>\n",
       "      <th>mean symmetry</th>\n",
       "      <th>mean fractal dimension</th>\n",
       "      <th>...</th>\n",
       "      <th>worst radius</th>\n",
       "      <th>worst texture</th>\n",
       "      <th>worst perimeter</th>\n",
       "      <th>worst area</th>\n",
       "      <th>worst smoothness</th>\n",
       "      <th>worst compactness</th>\n",
       "      <th>worst concavity</th>\n",
       "      <th>worst concave points</th>\n",
       "      <th>worst symmetry</th>\n",
       "      <th>worst fractal dimension</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>17.99</td>\n",
       "      <td>10.38</td>\n",
       "      <td>122.80</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>0.11840</td>\n",
       "      <td>0.27760</td>\n",
       "      <td>0.3001</td>\n",
       "      <td>0.14710</td>\n",
       "      <td>0.2419</td>\n",
       "      <td>0.07871</td>\n",
       "      <td>...</td>\n",
       "      <td>25.38</td>\n",
       "      <td>17.33</td>\n",
       "      <td>184.60</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>0.1622</td>\n",
       "      <td>0.6656</td>\n",
       "      <td>0.7119</td>\n",
       "      <td>0.2654</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.11890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20.57</td>\n",
       "      <td>17.77</td>\n",
       "      <td>132.90</td>\n",
       "      <td>1326.0</td>\n",
       "      <td>0.08474</td>\n",
       "      <td>0.07864</td>\n",
       "      <td>0.0869</td>\n",
       "      <td>0.07017</td>\n",
       "      <td>0.1812</td>\n",
       "      <td>0.05667</td>\n",
       "      <td>...</td>\n",
       "      <td>24.99</td>\n",
       "      <td>23.41</td>\n",
       "      <td>158.80</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>0.1238</td>\n",
       "      <td>0.1866</td>\n",
       "      <td>0.2416</td>\n",
       "      <td>0.1860</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>0.08902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>19.69</td>\n",
       "      <td>21.25</td>\n",
       "      <td>130.00</td>\n",
       "      <td>1203.0</td>\n",
       "      <td>0.10960</td>\n",
       "      <td>0.15990</td>\n",
       "      <td>0.1974</td>\n",
       "      <td>0.12790</td>\n",
       "      <td>0.2069</td>\n",
       "      <td>0.05999</td>\n",
       "      <td>...</td>\n",
       "      <td>23.57</td>\n",
       "      <td>25.53</td>\n",
       "      <td>152.50</td>\n",
       "      <td>1709.0</td>\n",
       "      <td>0.1444</td>\n",
       "      <td>0.4245</td>\n",
       "      <td>0.4504</td>\n",
       "      <td>0.2430</td>\n",
       "      <td>0.3613</td>\n",
       "      <td>0.08758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11.42</td>\n",
       "      <td>20.38</td>\n",
       "      <td>77.58</td>\n",
       "      <td>386.1</td>\n",
       "      <td>0.14250</td>\n",
       "      <td>0.28390</td>\n",
       "      <td>0.2414</td>\n",
       "      <td>0.10520</td>\n",
       "      <td>0.2597</td>\n",
       "      <td>0.09744</td>\n",
       "      <td>...</td>\n",
       "      <td>14.91</td>\n",
       "      <td>26.50</td>\n",
       "      <td>98.87</td>\n",
       "      <td>567.7</td>\n",
       "      <td>0.2098</td>\n",
       "      <td>0.8663</td>\n",
       "      <td>0.6869</td>\n",
       "      <td>0.2575</td>\n",
       "      <td>0.6638</td>\n",
       "      <td>0.17300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>20.29</td>\n",
       "      <td>14.34</td>\n",
       "      <td>135.10</td>\n",
       "      <td>1297.0</td>\n",
       "      <td>0.10030</td>\n",
       "      <td>0.13280</td>\n",
       "      <td>0.1980</td>\n",
       "      <td>0.10430</td>\n",
       "      <td>0.1809</td>\n",
       "      <td>0.05883</td>\n",
       "      <td>...</td>\n",
       "      <td>22.54</td>\n",
       "      <td>16.67</td>\n",
       "      <td>152.20</td>\n",
       "      <td>1575.0</td>\n",
       "      <td>0.1374</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>0.4000</td>\n",
       "      <td>0.1625</td>\n",
       "      <td>0.2364</td>\n",
       "      <td>0.07678</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 30 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean radius  mean texture  mean perimeter  mean area  mean smoothness  \\\n",
       "0        17.99         10.38          122.80     1001.0          0.11840   \n",
       "1        20.57         17.77          132.90     1326.0          0.08474   \n",
       "2        19.69         21.25          130.00     1203.0          0.10960   \n",
       "3        11.42         20.38           77.58      386.1          0.14250   \n",
       "4        20.29         14.34          135.10     1297.0          0.10030   \n",
       "\n",
       "   mean compactness  mean concavity  mean concave points  mean symmetry  \\\n",
       "0           0.27760          0.3001              0.14710         0.2419   \n",
       "1           0.07864          0.0869              0.07017         0.1812   \n",
       "2           0.15990          0.1974              0.12790         0.2069   \n",
       "3           0.28390          0.2414              0.10520         0.2597   \n",
       "4           0.13280          0.1980              0.10430         0.1809   \n",
       "\n",
       "   mean fractal dimension  ...  worst radius  worst texture  worst perimeter  \\\n",
       "0                 0.07871  ...         25.38          17.33           184.60   \n",
       "1                 0.05667  ...         24.99          23.41           158.80   \n",
       "2                 0.05999  ...         23.57          25.53           152.50   \n",
       "3                 0.09744  ...         14.91          26.50            98.87   \n",
       "4                 0.05883  ...         22.54          16.67           152.20   \n",
       "\n",
       "   worst area  worst smoothness  worst compactness  worst concavity  \\\n",
       "0      2019.0            0.1622             0.6656           0.7119   \n",
       "1      1956.0            0.1238             0.1866           0.2416   \n",
       "2      1709.0            0.1444             0.4245           0.4504   \n",
       "3       567.7            0.2098             0.8663           0.6869   \n",
       "4      1575.0            0.1374             0.2050           0.4000   \n",
       "\n",
       "   worst concave points  worst symmetry  worst fractal dimension  \n",
       "0                0.2654          0.4601                  0.11890  \n",
       "1                0.1860          0.2750                  0.08902  \n",
       "2                0.2430          0.3613                  0.08758  \n",
       "3                0.2575          0.6638                  0.17300  \n",
       "4                0.1625          0.2364                  0.07678  \n",
       "\n",
       "[5 rows x 30 columns]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "\n",
    "cancer = load_breast_cancer()\n",
    "df_X = pd.DataFrame(cancer['data'], columns=cancer['feature_names'])\n",
    "df_X.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9858c66",
   "metadata": {},
   "source": [
    "接下来为特征和目标标签创建`df_X`和`df_y`，如下所示："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "21d18a00",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Cancer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Cancer\n",
       "0       0\n",
       "1       0\n",
       "2       0\n",
       "3       0\n",
       "4       0"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_y = pd.DataFrame(cancer['target'], columns=['Cancer'])\n",
    "df_y.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fce020b",
   "metadata": {},
   "source": [
    "> PS ：如果想了解更多关于数据集的信息，可以运行`print(cancer['DESCR'])`打印出摘要和特征信息。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "22f22551",
   "metadata": {},
   "outputs": [],
   "source": [
    "# print(cancer['DESCR'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f07c227e",
   "metadata": {},
   "source": [
    "接下来，使用`training_test_split()`方法将数据集拆分为训练集 \\(70\\%\\) 和测试集 \\(30\\%\\) ："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ac1969eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# train test split\n",
    "from sklearn.model_selection import train_test_split\n",
    "import numpy as np\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(df_X,\n",
    "                                                    np.ravel(df_y),\n",
    "                                                    test_size=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9b9b725",
   "metadata": {},
   "source": [
    "我们将训练**支持向量分类器**\\(SVC\\) 模型。正则化参数`C`和核系数`gamma`是 SVC 中最重要的两个超参数：\n",
    "\n",
    "- 正则化参数`C`决定了正则化的强度。\n",
    "- 核系数`gamma`控制核的宽度。SVC 默认使用**径向基函数 \\(RBF\\)**核（也称为**高斯核**）。\n",
    "\n",
    "我们将在以下教程中调整这两个参数。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f82a6929",
   "metadata": {},
   "source": [
    "## 网格搜索"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10a155f5",
   "metadata": {},
   "source": [
    "最优值`C`和`gamma`是比较难找得到的。最简单的解决方案是尝试一堆组合，看看哪种组合效果最好。这种创建参数“网格”并尝试所有可能组合的方法称为**网格搜索。**\n",
    "\n",
    "![image.png](images/1.png)\n",
    "\n",
    "这种方法非常常见，所以 Scikit-learn 在`GridSearchCV`中内置了这种功能。CV 代表交叉验证，这是另一种评估和改进机器学习模型的技术。\n",
    "\n",
    "`GridSearchCV`需要一个描述准备尝试的参数和要训练的模型的字典。网格搜索的参数网格定义为字典，其中键是参数，值是要测试的一系列设置值。下面动手试试，首先定义候选参数` C``和``gamma `，如下所示："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f133b38b",
   "metadata": {},
   "outputs": [],
   "source": [
    "param_grid = { \n",
    "  'C': [0.1, 1, 10, 100, 1000], \n",
    "  'gamma': [1, 0.1, 0.01, 0.001, 0.0001] \n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ea8adea",
   "metadata": {},
   "source": [
    "接下来创建一个`GridSearchCV`对象，并使用训练数据进行训练模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e660eeaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.svm import SVC\n",
    "svc1=SVC()\n",
    "grid = GridSearchCV(svc1, param_grid, refit=True, verbose=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e843fb99",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 25 candidates, totalling 125 fits\n",
      "[CV] C=0.1, gamma=1 ..................................................\n",
      "[CV] ...................... C=0.1, gamma=1, score=0.600, total=   0.0s\n",
      "[CV] C=0.1, gamma=1 ..................................................\n",
      "[CV] ...................... C=0.1, gamma=1, score=0.600, total=   0.0s\n",
      "[CV] C=0.1, gamma=1 ..................................................\n",
      "[CV] ...................... C=0.1, gamma=1, score=0.600, total=   0.0s\n",
      "[CV] C=0.1, gamma=1 ..................................................\n",
      "[CV] ...................... C=0.1, gamma=1, score=0.595, total=   0.0s\n",
      "[CV] C=0.1, gamma=1 ..................................................\n",
      "[CV] ...................... C=0.1, gamma=1, score=0.608, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.1 ................................................\n",
      "[CV] .................... C=0.1, gamma=0.1, score=0.600, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.1 ................................................\n",
      "[CV] .................... C=0.1, gamma=0.1, score=0.600, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.1 ................................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .................... C=0.1, gamma=0.1, score=0.600, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.1 ................................................\n",
      "[CV] .................... C=0.1, gamma=0.1, score=0.595, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.1 ................................................\n",
      "[CV] .................... C=0.1, gamma=0.1, score=0.608, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.01 ...............................................\n",
      "[CV] ................... C=0.1, gamma=0.01, score=0.600, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.01 ...............................................\n",
      "[CV] ................... C=0.1, gamma=0.01, score=0.600, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.01 ...............................................\n",
      "[CV] ................... C=0.1, gamma=0.01, score=0.600, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.01 ...............................................\n",
      "[CV] ................... C=0.1, gamma=0.01, score=0.595, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.01 ...............................................\n",
      "[CV] ................... C=0.1, gamma=0.01, score=0.608, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.001 ..............................................\n",
      "[CV] .................. C=0.1, gamma=0.001, score=0.600, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.001 ..............................................\n",
      "[CV] .................. C=0.1, gamma=0.001, score=0.600, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.001 ..............................................\n",
      "[CV] .................. C=0.1, gamma=0.001, score=0.600, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.001 ..............................................\n",
      "[CV] .................. C=0.1, gamma=0.001, score=0.595, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.001 ..............................................\n",
      "[CV] .................. C=0.1, gamma=0.001, score=0.608, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.0001 .............................................\n",
      "[CV] ................. C=0.1, gamma=0.0001, score=0.963, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.0001 .............................................\n",
      "[CV] ................. C=0.1, gamma=0.0001, score=0.963, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.0001 .............................................\n",
      "[CV] ................. C=0.1, gamma=0.0001, score=0.887, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.0001 .............................................\n",
      "[CV] ................. C=0.1, gamma=0.0001, score=0.886, total=   0.0s\n",
      "[CV] C=0.1, gamma=0.0001 .............................................\n",
      "[CV] ................. C=0.1, gamma=0.0001, score=0.962, total=   0.0s\n",
      "[CV] C=1, gamma=1 ....................................................\n",
      "[CV] ........................ C=1, gamma=1, score=0.600, total=   0.0s\n",
      "[CV] C=1, gamma=1 ....................................................\n",
      "[CV] ........................ C=1, gamma=1, score=0.600, total=   0.0s\n",
      "[CV] C=1, gamma=1 ....................................................\n",
      "[CV] ........................ C=1, gamma=1, score=0.600, total=   0.0s\n",
      "[CV] C=1, gamma=1 ....................................................\n",
      "[CV] ........................ C=1, gamma=1, score=0.595, total=   0.0s\n",
      "[CV] C=1, gamma=1 ....................................................\n",
      "[CV] ........................ C=1, gamma=1, score=0.608, total=   0.0s\n",
      "[CV] C=1, gamma=0.1 ..................................................\n",
      "[CV] ...................... C=1, gamma=0.1, score=0.600, total=   0.0s\n",
      "[CV] C=1, gamma=0.1 ..................................................\n",
      "[CV] ...................... C=1, gamma=0.1, score=0.600, total=   0.0s\n",
      "[CV] C=1, gamma=0.1 ..................................................\n",
      "[CV] ...................... C=1, gamma=0.1, score=0.600, total=   0.0s\n",
      "[CV] C=1, gamma=0.1 ..................................................\n",
      "[CV] ...................... C=1, gamma=0.1, score=0.595, total=   0.0s\n",
      "[CV] C=1, gamma=0.1 ..................................................\n",
      "[CV] ...................... C=1, gamma=0.1, score=0.608, total=   0.0s\n",
      "[CV] C=1, gamma=0.01 .................................................\n",
      "[CV] ..................... C=1, gamma=0.01, score=0.613, total=   0.0s\n",
      "[CV] C=1, gamma=0.01 .................................................\n",
      "[CV] ..................... C=1, gamma=0.01, score=0.600, total=   0.0s\n",
      "[CV] C=1, gamma=0.01 .................................................\n",
      "[CV] ..................... C=1, gamma=0.01, score=0.613, total=   0.0s\n",
      "[CV] C=1, gamma=0.01 .................................................\n",
      "[CV] ..................... C=1, gamma=0.01, score=0.595, total=   0.0s\n",
      "[CV] C=1, gamma=0.01 .................................................\n",
      "[CV] ..................... C=1, gamma=0.01, score=0.608, total=   0.0s\n",
      "[CV] C=1, gamma=0.001 ................................................\n",
      "[CV] .................... C=1, gamma=0.001, score=0.950, total=   0.0s\n",
      "[CV] C=1, gamma=0.001 ................................................\n",
      "[CV] .................... C=1, gamma=0.001, score=0.938, total=   0.0s\n",
      "[CV] C=1, gamma=0.001 ................................................\n",
      "[CV] .................... C=1, gamma=0.001, score=0.900, total=   0.0s\n",
      "[CV] C=1, gamma=0.001 ................................................\n",
      "[CV] .................... C=1, gamma=0.001, score=0.899, total=   0.0s\n",
      "[CV] C=1, gamma=0.001 ................................................\n",
      "[CV] .................... C=1, gamma=0.001, score=0.924, total=   0.0s\n",
      "[CV] C=1, gamma=0.0001 ...............................................\n",
      "[CV] ................... C=1, gamma=0.0001, score=0.963, total=   0.0s\n",
      "[CV] C=1, gamma=0.0001 ...............................................\n",
      "[CV] ................... C=1, gamma=0.0001, score=0.988, total=   0.0s\n",
      "[CV] C=1, gamma=0.0001 ...............................................\n",
      "[CV] ................... C=1, gamma=0.0001, score=0.900, total=   0.0s\n",
      "[CV] C=1, gamma=0.0001 ...............................................\n",
      "[CV] ................... C=1, gamma=0.0001, score=0.873, total=   0.0s\n",
      "[CV] C=1, gamma=0.0001 ...............................................\n",
      "[CV] ................... C=1, gamma=0.0001, score=0.962, total=   0.0s\n",
      "[CV] C=10, gamma=1 ...................................................\n",
      "[CV] ....................... C=10, gamma=1, score=0.600, total=   0.0s\n",
      "[CV] C=10, gamma=1 ...................................................\n",
      "[CV] ....................... C=10, gamma=1, score=0.600, total=   0.0s\n",
      "[CV] C=10, gamma=1 ...................................................\n",
      "[CV] ....................... C=10, gamma=1, score=0.600, total=   0.0s\n",
      "[CV] C=10, gamma=1 ...................................................\n",
      "[CV] ....................... C=10, gamma=1, score=0.595, total=   0.0s\n",
      "[CV] C=10, gamma=1 ...................................................\n",
      "[CV] ....................... C=10, gamma=1, score=0.608, total=   0.0s\n",
      "[CV] C=10, gamma=0.1 .................................................\n",
      "[CV] ..................... C=10, gamma=0.1, score=0.600, total=   0.0s\n",
      "[CV] C=10, gamma=0.1 .................................................\n",
      "[CV] ..................... C=10, gamma=0.1, score=0.600, total=   0.0s\n",
      "[CV] C=10, gamma=0.1 .................................................\n",
      "[CV] ..................... C=10, gamma=0.1, score=0.600, total=   0.0s\n",
      "[CV] C=10, gamma=0.1 .................................................\n",
      "[CV] ..................... C=10, gamma=0.1, score=0.595, total=   0.0s\n",
      "[CV] C=10, gamma=0.1 .................................................\n",
      "[CV] ..................... C=10, gamma=0.1, score=0.608, total=   0.0s\n",
      "[CV] C=10, gamma=0.01 ................................................\n",
      "[CV] .................... C=10, gamma=0.01, score=0.625, total=   0.0s\n",
      "[CV] C=10, gamma=0.01 ................................................\n",
      "[CV] .................... C=10, gamma=0.01, score=0.600, total=   0.0s\n",
      "[CV] C=10, gamma=0.01 ................................................\n",
      "[CV] .................... C=10, gamma=0.01, score=0.613, total=   0.0s\n",
      "[CV] C=10, gamma=0.01 ................................................\n",
      "[CV] .................... C=10, gamma=0.01, score=0.608, total=   0.0s\n",
      "[CV] C=10, gamma=0.01 ................................................\n",
      "[CV] .................... C=10, gamma=0.01, score=0.608, total=   0.0s\n",
      "[CV] C=10, gamma=0.001 ...............................................\n",
      "[CV] ................... C=10, gamma=0.001, score=0.938, total=   0.0s\n",
      "[CV] C=10, gamma=0.001 ...............................................\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ................... C=10, gamma=0.001, score=0.950, total=   0.0s\n",
      "[CV] C=10, gamma=0.001 ...............................................\n",
      "[CV] ................... C=10, gamma=0.001, score=0.887, total=   0.0s\n",
      "[CV] C=10, gamma=0.001 ...............................................\n",
      "[CV] ................... C=10, gamma=0.001, score=0.886, total=   0.0s\n",
      "[CV] C=10, gamma=0.001 ...............................................\n",
      "[CV] ................... C=10, gamma=0.001, score=0.924, total=   0.0s\n",
      "[CV] C=10, gamma=0.0001 ..............................................\n",
      "[CV] .................. C=10, gamma=0.0001, score=0.963, total=   0.0s\n",
      "[CV] C=10, gamma=0.0001 ..............................................\n",
      "[CV] .................. C=10, gamma=0.0001, score=0.938, total=   0.0s\n",
      "[CV] C=10, gamma=0.0001 ..............................................\n",
      "[CV] .................. C=10, gamma=0.0001, score=0.900, total=   0.0s\n",
      "[CV] C=10, gamma=0.0001 ..............................................\n",
      "[CV] .................. C=10, gamma=0.0001, score=0.911, total=   0.0s\n",
      "[CV] C=10, gamma=0.0001 ..............................................\n",
      "[CV] .................. C=10, gamma=0.0001, score=0.924, total=   0.0s\n",
      "[CV] C=100, gamma=1 ..................................................\n",
      "[CV] ...................... C=100, gamma=1, score=0.600, total=   0.0s\n",
      "[CV] C=100, gamma=1 ..................................................\n",
      "[CV] ...................... C=100, gamma=1, score=0.600, total=   0.0s\n",
      "[CV] C=100, gamma=1 ..................................................\n",
      "[CV] ...................... C=100, gamma=1, score=0.600, total=   0.0s\n",
      "[CV] C=100, gamma=1 ..................................................\n",
      "[CV] ...................... C=100, gamma=1, score=0.595, total=   0.0s\n",
      "[CV] C=100, gamma=1 ..................................................\n",
      "[CV] ...................... C=100, gamma=1, score=0.608, total=   0.0s\n",
      "[CV] C=100, gamma=0.1 ................................................\n",
      "[CV] .................... C=100, gamma=0.1, score=0.600, total=   0.0s\n",
      "[CV] C=100, gamma=0.1 ................................................\n",
      "[CV] .................... C=100, gamma=0.1, score=0.600, total=   0.0s\n",
      "[CV] C=100, gamma=0.1 ................................................\n",
      "[CV] .................... C=100, gamma=0.1, score=0.600, total=   0.0s\n",
      "[CV] C=100, gamma=0.1 ................................................\n",
      "[CV] .................... C=100, gamma=0.1, score=0.595, total=   0.0s\n",
      "[CV] C=100, gamma=0.1 ................................................\n",
      "[CV] .................... C=100, gamma=0.1, score=0.608, total=   0.0s\n",
      "[CV] C=100, gamma=0.01 ...............................................\n",
      "[CV] ................... C=100, gamma=0.01, score=0.625, total=   0.0s\n",
      "[CV] C=100, gamma=0.01 ...............................................\n",
      "[CV] ................... C=100, gamma=0.01, score=0.600, total=   0.0s\n",
      "[CV] C=100, gamma=0.01 ...............................................\n",
      "[CV] ................... C=100, gamma=0.01, score=0.613, total=   0.0s\n",
      "[CV] C=100, gamma=0.01 ...............................................\n",
      "[CV] ................... C=100, gamma=0.01, score=0.608, total=   0.0s\n",
      "[CV] C=100, gamma=0.01 ...............................................\n",
      "[CV] ................... C=100, gamma=0.01, score=0.608, total=   0.0s\n",
      "[CV] C=100, gamma=0.001 ..............................................\n",
      "[CV] .................. C=100, gamma=0.001, score=0.938, total=   0.0s\n",
      "[CV] C=100, gamma=0.001 ..............................................\n",
      "[CV] .................. C=100, gamma=0.001, score=0.950, total=   0.0s\n",
      "[CV] C=100, gamma=0.001 ..............................................\n",
      "[CV] .................. C=100, gamma=0.001, score=0.887, total=   0.0s\n",
      "[CV] C=100, gamma=0.001 ..............................................\n",
      "[CV] .................. C=100, gamma=0.001, score=0.886, total=   0.0s\n",
      "[CV] C=100, gamma=0.001 ..............................................\n",
      "[CV] .................. C=100, gamma=0.001, score=0.924, total=   0.0s\n",
      "[CV] C=100, gamma=0.0001 .............................................\n",
      "[CV] ................. C=100, gamma=0.0001, score=0.950, total=   0.0s\n",
      "[CV] C=100, gamma=0.0001 .............................................\n",
      "[CV] ................. C=100, gamma=0.0001, score=0.950, total=   0.0s\n",
      "[CV] C=100, gamma=0.0001 .............................................\n",
      "[CV] ................. C=100, gamma=0.0001, score=0.912, total=   0.0s\n",
      "[CV] C=100, gamma=0.0001 .............................................\n",
      "[CV] ................. C=100, gamma=0.0001, score=0.911, total=   0.0s\n",
      "[CV] C=100, gamma=0.0001 .............................................\n",
      "[CV] ................. C=100, gamma=0.0001, score=0.924, total=   0.0s\n",
      "[CV] C=1000, gamma=1 .................................................\n",
      "[CV] ..................... C=1000, gamma=1, score=0.600, total=   0.0s\n",
      "[CV] C=1000, gamma=1 .................................................\n",
      "[CV] ..................... C=1000, gamma=1, score=0.600, total=   0.0s\n",
      "[CV] C=1000, gamma=1 .................................................\n",
      "[CV] ..................... C=1000, gamma=1, score=0.600, total=   0.0s\n",
      "[CV] C=1000, gamma=1 .................................................\n",
      "[CV] ..................... C=1000, gamma=1, score=0.595, total=   0.0s\n",
      "[CV] C=1000, gamma=1 .................................................\n",
      "[CV] ..................... C=1000, gamma=1, score=0.608, total=   0.0s\n",
      "[CV] C=1000, gamma=0.1 ...............................................\n",
      "[CV] ................... C=1000, gamma=0.1, score=0.600, total=   0.0s\n",
      "[CV] C=1000, gamma=0.1 ...............................................\n",
      "[CV] ................... C=1000, gamma=0.1, score=0.600, total=   0.0s\n",
      "[CV] C=1000, gamma=0.1 ...............................................\n",
      "[CV] ................... C=1000, gamma=0.1, score=0.600, total=   0.0s\n",
      "[CV] C=1000, gamma=0.1 ...............................................\n",
      "[CV] ................... C=1000, gamma=0.1, score=0.595, total=   0.0s\n",
      "[CV] C=1000, gamma=0.1 ...............................................\n",
      "[CV] ................... C=1000, gamma=0.1, score=0.608, total=   0.0s\n",
      "[CV] C=1000, gamma=0.01 ..............................................\n",
      "[CV] .................. C=1000, gamma=0.01, score=0.625, total=   0.0s\n",
      "[CV] C=1000, gamma=0.01 ..............................................\n",
      "[CV] .................. C=1000, gamma=0.01, score=0.600, total=   0.0s\n",
      "[CV] C=1000, gamma=0.01 ..............................................\n",
      "[CV] .................. C=1000, gamma=0.01, score=0.613, total=   0.0s\n",
      "[CV] C=1000, gamma=0.01 ..............................................\n",
      "[CV] .................. C=1000, gamma=0.01, score=0.608, total=   0.0s\n",
      "[CV] C=1000, gamma=0.01 ..............................................\n",
      "[CV] .................. C=1000, gamma=0.01, score=0.608, total=   0.0s\n",
      "[CV] C=1000, gamma=0.001 .............................................\n",
      "[CV] ................. C=1000, gamma=0.001, score=0.938, total=   0.0s\n",
      "[CV] C=1000, gamma=0.001 .............................................\n",
      "[CV] ................. C=1000, gamma=0.001, score=0.950, total=   0.0s\n",
      "[CV] C=1000, gamma=0.001 .............................................\n",
      "[CV] ................. C=1000, gamma=0.001, score=0.887, total=   0.0s\n",
      "[CV] C=1000, gamma=0.001 .............................................\n",
      "[CV] ................. C=1000, gamma=0.001, score=0.886, total=   0.0s\n",
      "[CV] C=1000, gamma=0.001 .............................................\n",
      "[CV] ................. C=1000, gamma=0.001, score=0.924, total=   0.0s\n",
      "[CV] C=1000, gamma=0.0001 ............................................\n",
      "[CV] ................ C=1000, gamma=0.0001, score=0.950, total=   0.0s\n",
      "[CV] C=1000, gamma=0.0001 ............................................\n",
      "[CV] ................ C=1000, gamma=0.0001, score=0.938, total=   0.0s\n",
      "[CV] C=1000, gamma=0.0001 ............................................\n",
      "[CV] ................ C=1000, gamma=0.0001, score=0.912, total=   0.0s\n",
      "[CV] C=1000, gamma=0.0001 ............................................\n",
      "[CV] ................ C=1000, gamma=0.0001, score=0.886, total=   0.0s\n",
      "[CV] C=1000, gamma=0.0001 ............................................\n",
      "[CV] ................ C=1000, gamma=0.0001, score=0.924, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done 125 out of 125 | elapsed:    2.6s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(estimator=SVC(),\n",
       "             param_grid={'C': [0.1, 1, 10, 100, 1000],\n",
       "                         'gamma': [1, 0.1, 0.01, 0.001, 0.0001]},\n",
       "             verbose=3)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9159ed85",
   "metadata": {},
   "source": [
    "一旦训练完成后，我们可以通过`GridSearchCV`的`best_params_`属性查看搜索到的最佳参数，并使用`best_estimator_`属性查看最佳模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b41334f3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'C': 1, 'gamma': 0.0001}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 找到最好的参数\n",
    "grid.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "39abb48a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=1, gamma=0.0001)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 找到最好的模型\n",
    "grid.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fd266b8",
   "metadata": {},
   "source": [
    "训练完成后，现在选择并采用该网格搜索到的最佳模型，并使用测试集进行预测并创建分类报告和混淆矩阵。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "829ae47e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 42  11]\n",
      " [  1 117]]\n"
     ]
    }
   ],
   "source": [
    "# 使用最好的估计器进行预测\n",
    "grid_predictions = grid.predict(X_test)\n",
    "# 混淆矩阵\n",
    "from sklearn.metrics import classification_report, confusion_matrix\n",
    "print(confusion_matrix(y_test, grid_predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e516e05e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.79      0.88        53\n",
      "           1       0.91      0.99      0.95       118\n",
      "\n",
      "    accuracy                           0.93       171\n",
      "   macro avg       0.95      0.89      0.91       171\n",
      "weighted avg       0.93      0.93      0.93       171\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 分类模型报告\n",
    "print(classification_report(y_test, grid_predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f60d201",
   "metadata": {},
   "source": [
    "## 随机搜索\n",
    "\n",
    "网格搜索尝试超参数的所有组合，因此增加了计算的时间复杂度，在数据量较大，或者模型较为复杂等等情况下，可能导致不可行的计算成本，这样网格搜索调参方法就不适用了。然而，**随机搜索**提供更便利的替代方案，该方法只测试你选择的超参数组成的元组，并且超参数值的选择是完全随机的，如下图所示。\n",
    "\n",
    "![随机搜索尝试随机组合](images/2.png)\n",
    "\n",
    "这种方法也很常见，所以 Scikit-learn 在`RandomizedSearchCV`中内置了这种功能。函数 API 与`GridSearchCV`类似。首先指定参数`C`和`gamma`以及参数值的候选样本的分布，如下所示:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "759d20d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "import scipy.stats as stats\n",
    "from sklearn.utils.fixes import loguniform\n",
    "\n",
    "# 指定采样的参数和分布\n",
    "param_dist = {\n",
    "  'C': stats.uniform(0.1, 1e4),\n",
    "  'gamma': loguniform(1e-6, 1e+1),\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82082bd8",
   "metadata": {},
   "source": [
    "接下来创建一个`RandomizedSearchCV`带参数`n_iter_search`的对象，并将使用训练数据来训练模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8cc207c5",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 20 candidates, totalling 100 fits\n",
      "[CV] C=1774.343244541459, gamma=3.1892778886544583e-06 ...............\n",
      "[CV]  C=1774.343244541459, gamma=3.1892778886544583e-06, score=0.988, total=   0.0s\n",
      "[CV] C=1774.343244541459, gamma=3.1892778886544583e-06 ...............\n",
      "[CV]  C=1774.343244541459, gamma=3.1892778886544583e-06, score=0.938, total=   0.0s\n",
      "[CV] C=1774.343244541459, gamma=3.1892778886544583e-06 ...............\n",
      "[CV]  C=1774.343244541459, gamma=3.1892778886544583e-06, score=0.938, total=   0.0s\n",
      "[CV] C=1774.343244541459, gamma=3.1892778886544583e-06 ...............\n",
      "[CV]  C=1774.343244541459, gamma=3.1892778886544583e-06, score=0.899, total=   0.0s\n",
      "[CV] C=1774.343244541459, gamma=3.1892778886544583e-06 ...............\n",
      "[CV]  C=1774.343244541459, gamma=3.1892778886544583e-06, score=0.962, total=   0.0s\n",
      "[CV] C=1503.2524541228202, gamma=0.10464256812063606 .................\n",
      "[CV]  C=1503.2524541228202, gamma=0.10464256812063606, score=0.600, total=   0.0s\n",
      "[CV] C=1503.2524541228202, gamma=0.10464256812063606 .................\n",
      "[CV]  C=1503.2524541228202, gamma=0.10464256812063606, score=0.600, total=   0.0s\n",
      "[CV] C=1503.2524541228202, gamma=0.10464256812063606 .................\n",
      "[CV]  C=1503.2524541228202, gamma=0.10464256812063606, score=0.600, total=   0.0s\n",
      "[CV] C=1503.2524541228202, gamma=0.10464256812063606 .................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=1503.2524541228202, gamma=0.10464256812063606, score=0.595, total=   0.0s\n",
      "[CV] C=1503.2524541228202, gamma=0.10464256812063606 .................\n",
      "[CV]  C=1503.2524541228202, gamma=0.10464256812063606, score=0.608, total=   0.0s\n",
      "[CV] C=1175.6858596772884, gamma=0.001591903045908838 ................\n",
      "[CV]  C=1175.6858596772884, gamma=0.001591903045908838, score=0.938, total=   0.0s\n",
      "[CV] C=1175.6858596772884, gamma=0.001591903045908838 ................\n",
      "[CV]  C=1175.6858596772884, gamma=0.001591903045908838, score=0.925, total=   0.0s\n",
      "[CV] C=1175.6858596772884, gamma=0.001591903045908838 ................\n",
      "[CV]  C=1175.6858596772884, gamma=0.001591903045908838, score=0.887, total=   0.0s\n",
      "[CV] C=1175.6858596772884, gamma=0.001591903045908838 ................\n",
      "[CV]  C=1175.6858596772884, gamma=0.001591903045908838, score=0.835, total=   0.0s\n",
      "[CV] C=1175.6858596772884, gamma=0.001591903045908838 ................\n",
      "[CV]  C=1175.6858596772884, gamma=0.001591903045908838, score=0.924, total=   0.0s\n",
      "[CV] C=9949.091253584631, gamma=0.09694064523745187 ..................\n",
      "[CV]  C=9949.091253584631, gamma=0.09694064523745187, score=0.600, total=   0.0s\n",
      "[CV] C=9949.091253584631, gamma=0.09694064523745187 ..................\n",
      "[CV]  C=9949.091253584631, gamma=0.09694064523745187, score=0.600, total=   0.0s\n",
      "[CV] C=9949.091253584631, gamma=0.09694064523745187 ..................\n",
      "[CV]  C=9949.091253584631, gamma=0.09694064523745187, score=0.600, total=   0.0s\n",
      "[CV] C=9949.091253584631, gamma=0.09694064523745187 ..................\n",
      "[CV]  C=9949.091253584631, gamma=0.09694064523745187, score=0.595, total=   0.0s\n",
      "[CV] C=9949.091253584631, gamma=0.09694064523745187 ..................\n",
      "[CV]  C=9949.091253584631, gamma=0.09694064523745187, score=0.608, total=   0.0s\n",
      "[CV] C=7062.033162389142, gamma=0.025314128570814526 .................\n",
      "[CV]  C=7062.033162389142, gamma=0.025314128570814526, score=0.600, total=   0.0s\n",
      "[CV] C=7062.033162389142, gamma=0.025314128570814526 .................\n",
      "[CV]  C=7062.033162389142, gamma=0.025314128570814526, score=0.600, total=   0.0s\n",
      "[CV] C=7062.033162389142, gamma=0.025314128570814526 .................\n",
      "[CV]  C=7062.033162389142, gamma=0.025314128570814526, score=0.600, total=   0.0s\n",
      "[CV] C=7062.033162389142, gamma=0.025314128570814526 .................\n",
      "[CV]  C=7062.033162389142, gamma=0.025314128570814526, score=0.595, total=   0.0s\n",
      "[CV] C=7062.033162389142, gamma=0.025314128570814526 .................\n",
      "[CV]  C=7062.033162389142, gamma=0.025314128570814526, score=0.608, total=   0.0s\n",
      "[CV] C=8645.579554506498, gamma=2.8458471754376826 ...................\n",
      "[CV]  C=8645.579554506498, gamma=2.8458471754376826, score=0.600, total=   0.0s\n",
      "[CV] C=8645.579554506498, gamma=2.8458471754376826 ...................\n",
      "[CV]  C=8645.579554506498, gamma=2.8458471754376826, score=0.600, total=   0.0s\n",
      "[CV] C=8645.579554506498, gamma=2.8458471754376826 ...................\n",
      "[CV]  C=8645.579554506498, gamma=2.8458471754376826, score=0.600, total=   0.0s\n",
      "[CV] C=8645.579554506498, gamma=2.8458471754376826 ...................\n",
      "[CV]  C=8645.579554506498, gamma=2.8458471754376826, score=0.595, total=   0.0s\n",
      "[CV] C=8645.579554506498, gamma=2.8458471754376826 ...................\n",
      "[CV]  C=8645.579554506498, gamma=2.8458471754376826, score=0.608, total=   0.0s\n",
      "[CV] C=9172.917553983025, gamma=0.22508011539786812 ..................\n",
      "[CV]  C=9172.917553983025, gamma=0.22508011539786812, score=0.600, total=   0.0s\n",
      "[CV] C=9172.917553983025, gamma=0.22508011539786812 ..................\n",
      "[CV]  C=9172.917553983025, gamma=0.22508011539786812, score=0.600, total=   0.0s\n",
      "[CV] C=9172.917553983025, gamma=0.22508011539786812 ..................\n",
      "[CV]  C=9172.917553983025, gamma=0.22508011539786812, score=0.600, total=   0.0s\n",
      "[CV] C=9172.917553983025, gamma=0.22508011539786812 ..................\n",
      "[CV]  C=9172.917553983025, gamma=0.22508011539786812, score=0.595, total=   0.0s\n",
      "[CV] C=9172.917553983025, gamma=0.22508011539786812 ..................\n",
      "[CV]  C=9172.917553983025, gamma=0.22508011539786812, score=0.608, total=   0.0s\n",
      "[CV] C=4688.375600254233, gamma=0.00019280688754327586 ...............\n",
      "[CV]  C=4688.375600254233, gamma=0.00019280688754327586, score=0.975, total=   0.0s\n",
      "[CV] C=4688.375600254233, gamma=0.00019280688754327586 ...............\n",
      "[CV]  C=4688.375600254233, gamma=0.00019280688754327586, score=0.925, total=   0.0s\n",
      "[CV] C=4688.375600254233, gamma=0.00019280688754327586 ...............\n",
      "[CV]  C=4688.375600254233, gamma=0.00019280688754327586, score=0.887, total=   0.0s\n",
      "[CV] C=4688.375600254233, gamma=0.00019280688754327586 ...............\n",
      "[CV]  C=4688.375600254233, gamma=0.00019280688754327586, score=0.899, total=   0.0s\n",
      "[CV] C=4688.375600254233, gamma=0.00019280688754327586 ...............\n",
      "[CV]  C=4688.375600254233, gamma=0.00019280688754327586, score=0.886, total=   0.0s\n",
      "[CV] C=5411.375675765696, gamma=0.2328071362790735 ...................\n",
      "[CV]  C=5411.375675765696, gamma=0.2328071362790735, score=0.600, total=   0.0s\n",
      "[CV] C=5411.375675765696, gamma=0.2328071362790735 ...................\n",
      "[CV]  C=5411.375675765696, gamma=0.2328071362790735, score=0.600, total=   0.0s\n",
      "[CV] C=5411.375675765696, gamma=0.2328071362790735 ...................\n",
      "[CV]  C=5411.375675765696, gamma=0.2328071362790735, score=0.600, total=   0.0s\n",
      "[CV] C=5411.375675765696, gamma=0.2328071362790735 ...................\n",
      "[CV]  C=5411.375675765696, gamma=0.2328071362790735, score=0.595, total=   0.0s\n",
      "[CV] C=5411.375675765696, gamma=0.2328071362790735 ...................\n",
      "[CV]  C=5411.375675765696, gamma=0.2328071362790735, score=0.608, total=   0.0s\n",
      "[CV] C=5348.859402328399, gamma=0.0001332402277403275 ................\n",
      "[CV]  C=5348.859402328399, gamma=0.0001332402277403275, score=0.963, total=   0.0s\n",
      "[CV] C=5348.859402328399, gamma=0.0001332402277403275 ................\n",
      "[CV]  C=5348.859402328399, gamma=0.0001332402277403275, score=0.912, total=   0.0s\n",
      "[CV] C=5348.859402328399, gamma=0.0001332402277403275 ................\n",
      "[CV]  C=5348.859402328399, gamma=0.0001332402277403275, score=0.912, total=   0.0s\n",
      "[CV] C=5348.859402328399, gamma=0.0001332402277403275 ................\n",
      "[CV]  C=5348.859402328399, gamma=0.0001332402277403275, score=0.899, total=   0.0s\n",
      "[CV] C=5348.859402328399, gamma=0.0001332402277403275 ................\n",
      "[CV]  C=5348.859402328399, gamma=0.0001332402277403275, score=0.899, total=   0.0s\n",
      "[CV] C=3306.4862531240483, gamma=8.8599919277554 .....................\n",
      "[CV]  C=3306.4862531240483, gamma=8.8599919277554, score=0.600, total=   0.0s\n",
      "[CV] C=3306.4862531240483, gamma=8.8599919277554 .....................\n",
      "[CV]  C=3306.4862531240483, gamma=8.8599919277554, score=0.600, total=   0.0s\n",
      "[CV] C=3306.4862531240483, gamma=8.8599919277554 .....................\n",
      "[CV]  C=3306.4862531240483, gamma=8.8599919277554, score=0.600, total=   0.0s\n",
      "[CV] C=3306.4862531240483, gamma=8.8599919277554 .....................\n",
      "[CV]  C=3306.4862531240483, gamma=8.8599919277554, score=0.595, total=   0.0s\n",
      "[CV] C=3306.4862531240483, gamma=8.8599919277554 .....................\n",
      "[CV]  C=3306.4862531240483, gamma=8.8599919277554, score=0.608, total=   0.0s\n",
      "[CV] C=6782.837501408192, gamma=0.0003840359158522522 ................\n",
      "[CV]  C=6782.837501408192, gamma=0.0003840359158522522, score=0.963, total=   0.0s\n",
      "[CV] C=6782.837501408192, gamma=0.0003840359158522522 ................\n",
      "[CV]  C=6782.837501408192, gamma=0.0003840359158522522, score=0.938, total=   0.0s\n",
      "[CV] C=6782.837501408192, gamma=0.0003840359158522522 ................\n",
      "[CV]  C=6782.837501408192, gamma=0.0003840359158522522, score=0.887, total=   0.0s\n",
      "[CV] C=6782.837501408192, gamma=0.0003840359158522522 ................\n",
      "[CV]  C=6782.837501408192, gamma=0.0003840359158522522, score=0.899, total=   0.0s\n",
      "[CV] C=6782.837501408192, gamma=0.0003840359158522522 ................\n",
      "[CV]  C=6782.837501408192, gamma=0.0003840359158522522, score=0.924, total=   0.0s\n",
      "[CV] C=8646.69312648182, gamma=0.0514422048188468 ....................\n",
      "[CV]  C=8646.69312648182, gamma=0.0514422048188468, score=0.600, total=   0.0s\n",
      "[CV] C=8646.69312648182, gamma=0.0514422048188468 ....................\n",
      "[CV]  C=8646.69312648182, gamma=0.0514422048188468, score=0.600, total=   0.0s\n",
      "[CV] C=8646.69312648182, gamma=0.0514422048188468 ....................\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=8646.69312648182, gamma=0.0514422048188468, score=0.600, total=   0.0s\n",
      "[CV] C=8646.69312648182, gamma=0.0514422048188468 ....................\n",
      "[CV]  C=8646.69312648182, gamma=0.0514422048188468, score=0.595, total=   0.0s\n",
      "[CV] C=8646.69312648182, gamma=0.0514422048188468 ....................\n",
      "[CV]  C=8646.69312648182, gamma=0.0514422048188468, score=0.608, total=   0.0s\n",
      "[CV] C=298.3475123803619, gamma=0.00010372525056028988 ...............\n",
      "[CV]  C=298.3475123803619, gamma=0.00010372525056028988, score=0.950, total=   0.0s\n",
      "[CV] C=298.3475123803619, gamma=0.00010372525056028988 ...............\n",
      "[CV]  C=298.3475123803619, gamma=0.00010372525056028988, score=0.938, total=   0.0s\n",
      "[CV] C=298.3475123803619, gamma=0.00010372525056028988 ...............\n",
      "[CV]  C=298.3475123803619, gamma=0.00010372525056028988, score=0.912, total=   0.0s\n",
      "[CV] C=298.3475123803619, gamma=0.00010372525056028988 ...............\n",
      "[CV]  C=298.3475123803619, gamma=0.00010372525056028988, score=0.911, total=   0.0s\n",
      "[CV] C=298.3475123803619, gamma=0.00010372525056028988 ...............\n",
      "[CV]  C=298.3475123803619, gamma=0.00010372525056028988, score=0.924, total=   0.0s\n",
      "[CV] C=8981.788000280414, gamma=1.7432010086816468e-05 ...............\n",
      "[CV]  C=8981.788000280414, gamma=1.7432010086816468e-05, score=0.938, total=   0.0s\n",
      "[CV] C=8981.788000280414, gamma=1.7432010086816468e-05 ...............\n",
      "[CV]  C=8981.788000280414, gamma=1.7432010086816468e-05, score=0.900, total=   0.0s\n",
      "[CV] C=8981.788000280414, gamma=1.7432010086816468e-05 ...............\n",
      "[CV]  C=8981.788000280414, gamma=1.7432010086816468e-05, score=0.938, total=   0.0s\n",
      "[CV] C=8981.788000280414, gamma=1.7432010086816468e-05 ...............\n",
      "[CV]  C=8981.788000280414, gamma=1.7432010086816468e-05, score=0.911, total=   0.0s\n",
      "[CV] C=8981.788000280414, gamma=1.7432010086816468e-05 ...............\n",
      "[CV]  C=8981.788000280414, gamma=1.7432010086816468e-05, score=0.911, total=   0.0s\n",
      "[CV] C=5280.585313970376, gamma=3.9370744645847915 ...................\n",
      "[CV]  C=5280.585313970376, gamma=3.9370744645847915, score=0.600, total=   0.0s\n",
      "[CV] C=5280.585313970376, gamma=3.9370744645847915 ...................\n",
      "[CV]  C=5280.585313970376, gamma=3.9370744645847915, score=0.600, total=   0.0s\n",
      "[CV] C=5280.585313970376, gamma=3.9370744645847915 ...................\n",
      "[CV]  C=5280.585313970376, gamma=3.9370744645847915, score=0.600, total=   0.0s\n",
      "[CV] C=5280.585313970376, gamma=3.9370744645847915 ...................\n",
      "[CV]  C=5280.585313970376, gamma=3.9370744645847915, score=0.595, total=   0.0s\n",
      "[CV] C=5280.585313970376, gamma=3.9370744645847915 ...................\n",
      "[CV]  C=5280.585313970376, gamma=3.9370744645847915, score=0.608, total=   0.0s\n",
      "[CV] C=1796.0060616565647, gamma=4.069550456489846 ...................\n",
      "[CV]  C=1796.0060616565647, gamma=4.069550456489846, score=0.600, total=   0.0s\n",
      "[CV] C=1796.0060616565647, gamma=4.069550456489846 ...................\n",
      "[CV]  C=1796.0060616565647, gamma=4.069550456489846, score=0.600, total=   0.0s\n",
      "[CV] C=1796.0060616565647, gamma=4.069550456489846 ...................\n",
      "[CV]  C=1796.0060616565647, gamma=4.069550456489846, score=0.600, total=   0.0s\n",
      "[CV] C=1796.0060616565647, gamma=4.069550456489846 ...................\n",
      "[CV]  C=1796.0060616565647, gamma=4.069550456489846, score=0.595, total=   0.0s\n",
      "[CV] C=1796.0060616565647, gamma=4.069550456489846 ...................\n",
      "[CV]  C=1796.0060616565647, gamma=4.069550456489846, score=0.608, total=   0.0s\n",
      "[CV] C=9364.034934800327, gamma=0.05309835444229084 ..................\n",
      "[CV]  C=9364.034934800327, gamma=0.05309835444229084, score=0.600, total=   0.0s\n",
      "[CV] C=9364.034934800327, gamma=0.05309835444229084 ..................\n",
      "[CV]  C=9364.034934800327, gamma=0.05309835444229084, score=0.600, total=   0.0s\n",
      "[CV] C=9364.034934800327, gamma=0.05309835444229084 ..................\n",
      "[CV]  C=9364.034934800327, gamma=0.05309835444229084, score=0.600, total=   0.0s\n",
      "[CV] C=9364.034934800327, gamma=0.05309835444229084 ..................\n",
      "[CV]  C=9364.034934800327, gamma=0.05309835444229084, score=0.595, total=   0.0s\n",
      "[CV] C=9364.034934800327, gamma=0.05309835444229084 ..................\n",
      "[CV]  C=9364.034934800327, gamma=0.05309835444229084, score=0.608, total=   0.0s\n",
      "[CV] C=1480.646519092858, gamma=3.768555228496674 ....................\n",
      "[CV]  C=1480.646519092858, gamma=3.768555228496674, score=0.600, total=   0.0s\n",
      "[CV] C=1480.646519092858, gamma=3.768555228496674 ....................\n",
      "[CV]  C=1480.646519092858, gamma=3.768555228496674, score=0.600, total=   0.0s\n",
      "[CV] C=1480.646519092858, gamma=3.768555228496674 ....................\n",
      "[CV]  C=1480.646519092858, gamma=3.768555228496674, score=0.600, total=   0.0s\n",
      "[CV] C=1480.646519092858, gamma=3.768555228496674 ....................\n",
      "[CV]  C=1480.646519092858, gamma=3.768555228496674, score=0.595, total=   0.0s\n",
      "[CV] C=1480.646519092858, gamma=3.768555228496674 ....................\n",
      "[CV]  C=1480.646519092858, gamma=3.768555228496674, score=0.608, total=   0.0s\n",
      "[CV] C=4859.453665124743, gamma=1.5771735759385928e-05 ...............\n",
      "[CV]  C=4859.453665124743, gamma=1.5771735759385928e-05, score=0.950, total=   0.0s\n",
      "[CV] C=4859.453665124743, gamma=1.5771735759385928e-05 ...............\n",
      "[CV]  C=4859.453665124743, gamma=1.5771735759385928e-05, score=0.912, total=   0.0s\n",
      "[CV] C=4859.453665124743, gamma=1.5771735759385928e-05 ...............\n",
      "[CV]  C=4859.453665124743, gamma=1.5771735759385928e-05, score=0.938, total=   0.0s\n",
      "[CV] C=4859.453665124743, gamma=1.5771735759385928e-05 ...............\n",
      "[CV]  C=4859.453665124743, gamma=1.5771735759385928e-05, score=0.911, total=   0.0s\n",
      "[CV] C=4859.453665124743, gamma=1.5771735759385928e-05 ...............\n",
      "[CV]  C=4859.453665124743, gamma=1.5771735759385928e-05, score=0.911, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done 100 out of 100 | elapsed:    1.9s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(estimator=SVC(), n_iter=20,\n",
       "                   param_distributions={'C': <scipy.stats._distn_infrastructure.rv_frozen object at 0x0000025DC039ED60>,\n",
       "                                        'gamma': <scipy.stats._distn_infrastructure.rv_frozen object at 0x0000025DC039E2B0>},\n",
       "                   verbose=3)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_iter_search = 20\n",
    "random_search = RandomizedSearchCV(SVC(),\n",
    "                                   param_distributions=param_dist,\n",
    "                                   n_iter=n_iter_search,\n",
    "                                   refit=True,\n",
    "                                   verbose=3)\n",
    "random_search.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85bcee0b",
   "metadata": {},
   "source": [
    "同样，一旦训练完成后，我们可以通过`RandomizedSearchCV`的`best_params_`属性查看搜索到的最佳参数，并使用`best_estimator_`属性查看得到的最佳模型："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28cb102d",
   "metadata": {},
   "source": [
    "预测 RandomizedSearchCV 并创建报告。\n",
    "\n",
    "最后，我们采用最终确定的最佳随机搜索模型，并使用测试集进行预测，并创建分类报告和混淆矩阵查看模型效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "78c50204",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 45   8]\n",
      " [  2 116]]\n"
     ]
    }
   ],
   "source": [
    "# 使用最好的估计器进行预测\n",
    "random_predictions = random_search.predict(X_test)\n",
    "\n",
    "from sklearn.metrics import classification_report, confusion_matrix\n",
    "# Confusion matrics\n",
    "print(confusion_matrix(y_test, random_predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "6cfff439",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      0.85      0.90        53\n",
      "           1       0.94      0.98      0.96       118\n",
      "\n",
      "    accuracy                           0.94       171\n",
      "   macro avg       0.95      0.92      0.93       171\n",
      "weighted avg       0.94      0.94      0.94       171\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 分类评价报告\n",
    "print(classification_report(y_test, random_predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79ea2f3c",
   "metadata": {},
   "source": [
    "## 贝叶斯搜索\n",
    "\n",
    "贝叶斯搜索使用贝叶斯优化技术对搜索空间进行建模，以尽快获得优化的参数值。它使用搜索空间的结构来优化搜索时间。贝叶斯搜索方法使用过去的评估结果来采样最有可能提供更好结果的新候选参数（如下图所示）。\n",
    "\n",
    "![贝叶斯搜索](images/3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a05f83c",
   "metadata": {},
   "source": [
    "[Scikit-Optimize](undefined \"undefined\")库带有 BayesSearchCV 实现。\n",
    "\n",
    "首先指定参数 C 和 gamma 以及参数值的候选样本的分布，如下所示:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "59b43872",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install scikit-optimize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "15205836",
   "metadata": {},
   "outputs": [],
   "source": [
    "from skopt import BayesSearchCV\n",
    "# 参数范围由下面的一个指定\n",
    "from skopt.space import Real, Categorical, Integer\n",
    "\n",
    "search_spaces = {\n",
    "    'C': Real(0.1, 1e+4),\n",
    "    'gamma': Real(1e-6, 1e+1, 'log-uniform'),\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69fd861a",
   "metadata": {},
   "source": [
    "接下来创建一个 BayesSearchCV 带参数 n_iter_search 的对象，并将使用训练数据来训练模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "34c048c0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=4977.397744560771, gamma=0.023829160053700667 .................\n",
      "[CV]  C=4977.397744560771, gamma=0.023829160053700667, score=0.600, total=   0.0s\n",
      "[CV] C=4977.397744560771, gamma=0.023829160053700667 .................\n",
      "[CV]  C=4977.397744560771, gamma=0.023829160053700667, score=0.600, total=   0.0s\n",
      "[CV] C=4977.397744560771, gamma=0.023829160053700667 .................\n",
      "[CV]  C=4977.397744560771, gamma=0.023829160053700667, score=0.600, total=   0.0s\n",
      "[CV] C=4977.397744560771, gamma=0.023829160053700667 .................\n",
      "[CV]  C=4977.397744560771, gamma=0.023829160053700667, score=0.595, total=   0.0s\n",
      "[CV] C=4977.397744560771, gamma=0.023829160053700667 .................\n",
      "[CV]  C=4977.397744560771, gamma=0.023829160053700667, score=0.608, total=   0.0s\n",
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=8758.852767384362, gamma=4.852350395777465 ....................\n",
      "[CV]  C=8758.852767384362, gamma=4.852350395777465, score=0.600, total=   0.0s\n",
      "[CV] C=8758.852767384362, gamma=4.852350395777465 ....................\n",
      "[CV]  C=8758.852767384362, gamma=4.852350395777465, score=0.600, total=   0.0s\n",
      "[CV] C=8758.852767384362, gamma=4.852350395777465 ....................\n",
      "[CV]  C=8758.852767384362, gamma=4.852350395777465, score=0.600, total=   0.0s\n",
      "[CV] C=8758.852767384362, gamma=4.852350395777465 ....................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=8758.852767384362, gamma=4.852350395777465, score=0.595, total=   0.0s\n",
      "[CV] C=8758.852767384362, gamma=4.852350395777465 ....................\n",
      "[CV]  C=8758.852767384362, gamma=4.852350395777465, score=0.608, total=   0.0s\n",
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=2910.2258495677825, gamma=0.9018344522099031 ..................\n",
      "[CV]  C=2910.2258495677825, gamma=0.9018344522099031, score=0.600, total=   0.0s\n",
      "[CV] C=2910.2258495677825, gamma=0.9018344522099031 ..................\n",
      "[CV]  C=2910.2258495677825, gamma=0.9018344522099031, score=0.600, total=   0.0s\n",
      "[CV] C=2910.2258495677825, gamma=0.9018344522099031 ..................\n",
      "[CV]  C=2910.2258495677825, gamma=0.9018344522099031, score=0.600, total=   0.0s\n",
      "[CV] C=2910.2258495677825, gamma=0.9018344522099031 ..................\n",
      "[CV]  C=2910.2258495677825, gamma=0.9018344522099031, score=0.595, total=   0.0s\n",
      "[CV] C=2910.2258495677825, gamma=0.9018344522099031 ..................\n",
      "[CV]  C=2910.2258495677825, gamma=0.9018344522099031, score=0.608, total=   0.0s\n",
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[CV] C=6201.076450375709, gamma=4.284329825969826e-06 ................\n",
      "[CV]  C=6201.076450375709, gamma=4.284329825969826e-06, score=0.988, total=   0.0s\n",
      "[CV] C=6201.076450375709, gamma=4.284329825969826e-06 ................\n",
      "[CV]  C=6201.076450375709, gamma=4.284329825969826e-06, score=0.925, total=   0.0s\n",
      "[CV] C=6201.076450375709, gamma=4.284329825969826e-06 ................\n",
      "[CV]  C=6201.076450375709, gamma=4.284329825969826e-06, score=0.938, total=   0.0s\n",
      "[CV] C=6201.076450375709, gamma=4.284329825969826e-06 ................\n",
      "[CV]  C=6201.076450375709, gamma=4.284329825969826e-06, score=0.924, total=   0.0s\n",
      "[CV] C=6201.076450375709, gamma=4.284329825969826e-06 ................\n",
      "[CV]  C=6201.076450375709, gamma=4.284329825969826e-06, score=0.949, total=   0.0s\n",
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=8519.566314535297, gamma=5.566534140000436 ....................\n",
      "[CV]  C=8519.566314535297, gamma=5.566534140000436, score=0.600, total=   0.0s\n",
      "[CV] C=8519.566314535297, gamma=5.566534140000436 ....................\n",
      "[CV]  C=8519.566314535297, gamma=5.566534140000436, score=0.600, total=   0.0s\n",
      "[CV] C=8519.566314535297, gamma=5.566534140000436 ....................\n",
      "[CV]  C=8519.566314535297, gamma=5.566534140000436, score=0.600, total=   0.0s\n",
      "[CV] C=8519.566314535297, gamma=5.566534140000436 ....................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=8519.566314535297, gamma=5.566534140000436, score=0.595, total=   0.0s\n",
      "[CV] C=8519.566314535297, gamma=5.566534140000436 ....................\n",
      "[CV]  C=8519.566314535297, gamma=5.566534140000436, score=0.608, total=   0.0s\n",
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=4992.078124786961, gamma=1.7807839452844108 ...................\n",
      "[CV]  C=4992.078124786961, gamma=1.7807839452844108, score=0.600, total=   0.0s\n",
      "[CV] C=4992.078124786961, gamma=1.7807839452844108 ...................\n",
      "[CV]  C=4992.078124786961, gamma=1.7807839452844108, score=0.600, total=   0.0s\n",
      "[CV] C=4992.078124786961, gamma=1.7807839452844108 ...................\n",
      "[CV]  C=4992.078124786961, gamma=1.7807839452844108, score=0.600, total=   0.0s\n",
      "[CV] C=4992.078124786961, gamma=1.7807839452844108 ...................\n",
      "[CV]  C=4992.078124786961, gamma=1.7807839452844108, score=0.595, total=   0.0s\n",
      "[CV] C=4992.078124786961, gamma=1.7807839452844108 ...................\n",
      "[CV]  C=4992.078124786961, gamma=1.7807839452844108, score=0.608, total=   0.0s\n",
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=9053.010519680269, gamma=0.11986519331682337 ..................\n",
      "[CV]  C=9053.010519680269, gamma=0.11986519331682337, score=0.600, total=   0.0s\n",
      "[CV] C=9053.010519680269, gamma=0.11986519331682337 ..................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  C=9053.010519680269, gamma=0.11986519331682337, score=0.600, total=   0.0s\n",
      "[CV] C=9053.010519680269, gamma=0.11986519331682337 ..................\n",
      "[CV]  C=9053.010519680269, gamma=0.11986519331682337, score=0.600, total=   0.0s\n",
      "[CV] C=9053.010519680269, gamma=0.11986519331682337 ..................\n",
      "[CV]  C=9053.010519680269, gamma=0.11986519331682337, score=0.595, total=   0.0s\n",
      "[CV] C=9053.010519680269, gamma=0.11986519331682337 ..................\n",
      "[CV]  C=9053.010519680269, gamma=0.11986519331682337, score=0.608, total=   0.0s\n",
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=8641.531299019009, gamma=0.07861618342270658 ..................\n",
      "[CV]  C=8641.531299019009, gamma=0.07861618342270658, score=0.600, total=   0.0s\n",
      "[CV] C=8641.531299019009, gamma=0.07861618342270658 ..................\n",
      "[CV]  C=8641.531299019009, gamma=0.07861618342270658, score=0.600, total=   0.0s\n",
      "[CV] C=8641.531299019009, gamma=0.07861618342270658 ..................\n",
      "[CV]  C=8641.531299019009, gamma=0.07861618342270658, score=0.600, total=   0.0s\n",
      "[CV] C=8641.531299019009, gamma=0.07861618342270658 ..................\n",
      "[CV]  C=8641.531299019009, gamma=0.07861618342270658, score=0.595, total=   0.0s\n",
      "[CV] C=8641.531299019009, gamma=0.07861618342270658 ..................\n",
      "[CV]  C=8641.531299019009, gamma=0.07861618342270658, score=0.608, total=   0.0s\n",
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[CV] C=1959.6879176103353, gamma=0.47778823806501924 .................\n",
      "[CV]  C=1959.6879176103353, gamma=0.47778823806501924, score=0.600, total=   0.0s\n",
      "[CV] C=1959.6879176103353, gamma=0.47778823806501924 .................\n",
      "[CV]  C=1959.6879176103353, gamma=0.47778823806501924, score=0.600, total=   0.0s\n",
      "[CV] C=1959.6879176103353, gamma=0.47778823806501924 .................\n",
      "[CV]  C=1959.6879176103353, gamma=0.47778823806501924, score=0.600, total=   0.0s\n",
      "[CV] C=1959.6879176103353, gamma=0.47778823806501924 .................\n",
      "[CV]  C=1959.6879176103353, gamma=0.47778823806501924, score=0.595, total=   0.0s\n",
      "[CV] C=1959.6879176103353, gamma=0.47778823806501924 .................\n",
      "[CV]  C=1959.6879176103353, gamma=0.47778823806501924, score=0.608, total=   0.0s\n",
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=5084.272437915128, gamma=0.09627687266838895 ..................\n",
      "[CV]  C=5084.272437915128, gamma=0.09627687266838895, score=0.600, total=   0.0s\n",
      "[CV] C=5084.272437915128, gamma=0.09627687266838895 ..................\n",
      "[CV]  C=5084.272437915128, gamma=0.09627687266838895, score=0.600, total=   0.0s\n",
      "[CV] C=5084.272437915128, gamma=0.09627687266838895 ..................\n",
      "[CV]  C=5084.272437915128, gamma=0.09627687266838895, score=0.600, total=   0.0s\n",
      "[CV] C=5084.272437915128, gamma=0.09627687266838895 ..................\n",
      "[CV]  C=5084.272437915128, gamma=0.09627687266838895, score=0.595, total=   0.0s\n",
      "[CV] C=5084.272437915128, gamma=0.09627687266838895 ..................\n",
      "[CV]  C=5084.272437915128, gamma=0.09627687266838895, score=0.608, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=10000.0, gamma=1e-06 ..........................................\n",
      "[CV] .............. C=10000.0, gamma=1e-06, score=0.988, total=   0.0s\n",
      "[CV] C=10000.0, gamma=1e-06 ..........................................\n",
      "[CV] .............. C=10000.0, gamma=1e-06, score=0.938, total=   0.0s\n",
      "[CV] C=10000.0, gamma=1e-06 ..........................................\n",
      "[CV] .............. C=10000.0, gamma=1e-06, score=0.938, total=   0.0s\n",
      "[CV] C=10000.0, gamma=1e-06 ..........................................\n",
      "[CV] .............. C=10000.0, gamma=1e-06, score=0.937, total=   0.0s\n",
      "[CV] C=10000.0, gamma=1e-06 ..........................................\n",
      "[CV] .............. C=10000.0, gamma=1e-06, score=0.975, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=0.1, gamma=1.4905928536395696e-06 .............................\n",
      "[CV] . C=0.1, gamma=1.4905928536395696e-06, score=0.925, total=   0.0s\n",
      "[CV] C=0.1, gamma=1.4905928536395696e-06 .............................\n",
      "[CV] . C=0.1, gamma=1.4905928536395696e-06, score=0.950, total=   0.0s\n",
      "[CV] C=0.1, gamma=1.4905928536395696e-06 .............................\n",
      "[CV] . C=0.1, gamma=1.4905928536395696e-06, score=0.863, total=   0.0s\n",
      "[CV] C=0.1, gamma=1.4905928536395696e-06 .............................\n",
      "[CV] . C=0.1, gamma=1.4905928536395696e-06, score=0.823, total=   0.0s\n",
      "[CV] C=0.1, gamma=1.4905928536395696e-06 .............................\n",
      "[CV] . C=0.1, gamma=1.4905928536395696e-06, score=0.949, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=10000.0, gamma=3.152182182099272e-06 ..........................\n",
      "[CV]  C=10000.0, gamma=3.152182182099272e-06, score=0.988, total=   0.0s\n",
      "[CV] C=10000.0, gamma=3.152182182099272e-06 ..........................\n",
      "[CV]  C=10000.0, gamma=3.152182182099272e-06, score=0.938, total=   0.0s\n",
      "[CV] C=10000.0, gamma=3.152182182099272e-06 ..........................\n",
      "[CV]  C=10000.0, gamma=3.152182182099272e-06, score=0.938, total=   0.0s\n",
      "[CV] C=10000.0, gamma=3.152182182099272e-06 ..........................\n",
      "[CV]  C=10000.0, gamma=3.152182182099272e-06, score=0.924, total=   0.0s\n",
      "[CV] C=10000.0, gamma=3.152182182099272e-06 ..........................\n",
      "[CV]  C=10000.0, gamma=3.152182182099272e-06, score=0.975, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=8258.305079914611, gamma=1.618411578782942e-06 ................\n",
      "[CV]  C=8258.305079914611, gamma=1.618411578782942e-06, score=0.988, total=   0.0s\n",
      "[CV] C=8258.305079914611, gamma=1.618411578782942e-06 ................\n",
      "[CV]  C=8258.305079914611, gamma=1.618411578782942e-06, score=0.938, total=   0.0s\n",
      "[CV] C=8258.305079914611, gamma=1.618411578782942e-06 ................\n",
      "[CV]  C=8258.305079914611, gamma=1.618411578782942e-06, score=0.938, total=   0.0s\n",
      "[CV] C=8258.305079914611, gamma=1.618411578782942e-06 ................\n",
      "[CV]  C=8258.305079914611, gamma=1.618411578782942e-06, score=0.924, total=   0.0s\n",
      "[CV] C=8258.305079914611, gamma=1.618411578782942e-06 ................\n",
      "[CV]  C=8258.305079914611, gamma=1.618411578782942e-06, score=0.962, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=10000.0, gamma=1.56539453002737e-06 ...........................\n",
      "[CV]  C=10000.0, gamma=1.56539453002737e-06, score=0.988, total=   0.0s\n",
      "[CV] C=10000.0, gamma=1.56539453002737e-06 ...........................\n",
      "[CV]  C=10000.0, gamma=1.56539453002737e-06, score=0.938, total=   0.0s\n",
      "[CV] C=10000.0, gamma=1.56539453002737e-06 ...........................\n",
      "[CV]  C=10000.0, gamma=1.56539453002737e-06, score=0.938, total=   0.0s\n",
      "[CV] C=10000.0, gamma=1.56539453002737e-06 ...........................\n",
      "[CV]  C=10000.0, gamma=1.56539453002737e-06, score=0.911, total=   0.0s\n",
      "[CV] C=10000.0, gamma=1.56539453002737e-06 ...........................\n",
      "[CV]  C=10000.0, gamma=1.56539453002737e-06, score=0.962, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=10000.0, gamma=1.3871576029368295e-05 .........................\n",
      "[CV]  C=10000.0, gamma=1.3871576029368295e-05, score=0.938, total=   0.0s\n",
      "[CV] C=10000.0, gamma=1.3871576029368295e-05 .........................\n",
      "[CV]  C=10000.0, gamma=1.3871576029368295e-05, score=0.900, total=   0.0s\n",
      "[CV] C=10000.0, gamma=1.3871576029368295e-05 .........................\n",
      "[CV]  C=10000.0, gamma=1.3871576029368295e-05, score=0.938, total=   0.0s\n",
      "[CV] C=10000.0, gamma=1.3871576029368295e-05 .........................\n",
      "[CV]  C=10000.0, gamma=1.3871576029368295e-05, score=0.911, total=   0.0s\n",
      "[CV] C=10000.0, gamma=1.3871576029368295e-05 .........................\n",
      "[CV]  C=10000.0, gamma=1.3871576029368295e-05, score=0.924, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=57.93860986876622, gamma=2.0990813445227366e-05 ...............\n",
      "[CV]  C=57.93860986876622, gamma=2.0990813445227366e-05, score=0.975, total=   0.0s\n",
      "[CV] C=57.93860986876622, gamma=2.0990813445227366e-05 ...............\n",
      "[CV]  C=57.93860986876622, gamma=2.0990813445227366e-05, score=0.925, total=   0.0s\n",
      "[CV] C=57.93860986876622, gamma=2.0990813445227366e-05 ...............\n",
      "[CV]  C=57.93860986876622, gamma=2.0990813445227366e-05, score=0.912, total=   0.0s\n",
      "[CV] C=57.93860986876622, gamma=2.0990813445227366e-05 ...............\n",
      "[CV]  C=57.93860986876622, gamma=2.0990813445227366e-05, score=0.911, total=   0.0s\n",
      "[CV] C=57.93860986876622, gamma=2.0990813445227366e-05 ...............\n",
      "[CV]  C=57.93860986876622, gamma=2.0990813445227366e-05, score=0.937, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=164.1340366211841, gamma=0.00010481680034418265 ...............\n",
      "[CV]  C=164.1340366211841, gamma=0.00010481680034418265, score=0.950, total=   0.0s\n",
      "[CV] C=164.1340366211841, gamma=0.00010481680034418265 ...............\n",
      "[CV]  C=164.1340366211841, gamma=0.00010481680034418265, score=0.950, total=   0.0s\n",
      "[CV] C=164.1340366211841, gamma=0.00010481680034418265 ...............\n",
      "[CV]  C=164.1340366211841, gamma=0.00010481680034418265, score=0.912, total=   0.0s\n",
      "[CV] C=164.1340366211841, gamma=0.00010481680034418265 ...............\n",
      "[CV]  C=164.1340366211841, gamma=0.00010481680034418265, score=0.911, total=   0.0s\n",
      "[CV] C=164.1340366211841, gamma=0.00010481680034418265 ...............\n",
      "[CV]  C=164.1340366211841, gamma=0.00010481680034418265, score=0.911, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=9914.701087332833, gamma=0.00013060755725868254 ...............\n",
      "[CV]  C=9914.701087332833, gamma=0.00013060755725868254, score=0.963, total=   0.0s\n",
      "[CV] C=9914.701087332833, gamma=0.00013060755725868254 ...............\n",
      "[CV]  C=9914.701087332833, gamma=0.00013060755725868254, score=0.912, total=   0.0s\n",
      "[CV] C=9914.701087332833, gamma=0.00013060755725868254 ...............\n",
      "[CV]  C=9914.701087332833, gamma=0.00013060755725868254, score=0.912, total=   0.0s\n",
      "[CV] C=9914.701087332833, gamma=0.00013060755725868254 ...............\n",
      "[CV]  C=9914.701087332833, gamma=0.00013060755725868254, score=0.899, total=   0.0s\n",
      "[CV] C=9914.701087332833, gamma=0.00013060755725868254 ...............\n",
      "[CV]  C=9914.701087332833, gamma=0.00013060755725868254, score=0.899, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n",
      "[CV] C=34.78128435599578, gamma=0.00046475689572079325 ...............\n",
      "[CV]  C=34.78128435599578, gamma=0.00046475689572079325, score=0.963, total=   0.0s\n",
      "[CV] C=34.78128435599578, gamma=0.00046475689572079325 ...............\n",
      "[CV]  C=34.78128435599578, gamma=0.00046475689572079325, score=0.938, total=   0.0s\n",
      "[CV] C=34.78128435599578, gamma=0.00046475689572079325 ...............\n",
      "[CV]  C=34.78128435599578, gamma=0.00046475689572079325, score=0.875, total=   0.0s\n",
      "[CV] C=34.78128435599578, gamma=0.00046475689572079325 ...............\n",
      "[CV]  C=34.78128435599578, gamma=0.00046475689572079325, score=0.886, total=   0.0s\n",
      "[CV] C=34.78128435599578, gamma=0.00046475689572079325 ...............\n",
      "[CV]  C=34.78128435599578, gamma=0.00046475689572079325, score=0.924, total=   0.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   5 out of   5 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "BayesSearchCV(cv=5, estimator=SVC(), n_iter=20,\n",
       "              search_spaces={'C': Real(low=0.1, high=10000.0, prior='uniform', transform='normalize'),\n",
       "                             'gamma': Real(low=1e-06, high=10.0, prior='log-uniform', transform='normalize')},\n",
       "              verbose=3)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_iter_search = 20\n",
    "bayes_search = BayesSearchCV(SVC(),\n",
    "                             search_spaces,\n",
    "                             n_iter=n_iter_search,\n",
    "                             cv=5,\n",
    "                             verbose=3)\n",
    "bayes_search.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fde80ed7",
   "metadata": {},
   "source": [
    "同样，一旦训练完成后，我们可以通过检查发现的最佳参数`BayesSearchCV`的`best_params_`属性，并在最佳估计`best_estimator_`属性："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "535b2598",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('C', 10000.0), ('gamma', 1e-06)])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bayes_search.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "dbe7aae4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=10000.0, gamma=1e-06)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bayes_search.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8baf1839",
   "metadata": {},
   "source": [
    "最后，我们采用贝叶斯搜索模型并使用测试集创建一些预测，并为它们创建分类报告和混淆矩阵。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "33737a41",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 46   7]\n",
      " [  1 117]]\n"
     ]
    }
   ],
   "source": [
    "bayes_predictions = bayes_search.predict(X_test)\n",
    "from sklearn.metrics import classification_report,confusion_matrix\n",
    "# 混淆矩阵\n",
    "print(confusion_matrix(y_test, bayes_predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "f94ba35c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.87      0.92        53\n",
      "           1       0.94      0.99      0.97       118\n",
      "\n",
      "    accuracy                           0.95       171\n",
      "   macro avg       0.96      0.93      0.94       171\n",
      "weighted avg       0.95      0.95      0.95       171\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 分类评价报告\n",
    "print(classification_report(y_test, bayes_predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3754a7d",
   "metadata": {},
   "source": [
    "## 写在最后\n",
    "\n",
    "\n",
    "在本文中，我们介绍了 3 种最流行的超参数优化技术，这些技术用于获得最佳超参数集，从而训练稳健的机器学习模型。\n",
    "\n",
    "一般来说，如果组合的数量足够有限，我们可以使用**网格搜索**技术。但是当组合数量增加时，我们应该尝试**随机搜索**或**贝叶斯搜索**，因为它们在计算上并不非常消耗资源。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
